Skip to content

Read the process table in one place, and stop spawning ps on Linux - #171

Merged
myobie merged 3 commits into
kill-escalatesfrom
one-process-table
Sep 3, 2026
Merged

Read the process table in one place, and stop spawning ps on Linux#171
myobie merged 3 commits into
kill-escalatesfrom
one-process-table

Conversation

@myobie

@myobie myobie commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #170. Base is kill-escalates, so the diff here is the
process table alone. The matching Rust change is
compoundingtech/pty-rust#8.

The question this answers

Nathan asked: "Should we just stop using ps or is the current code alright?"

The current code is not alright, and the reason is arithmetic.

The teardown polls every 25 ms and asked the operating system about each
surviving descendant separately — a ps spawn each time. Inside the 1500 ms
TERM budget that is up to 60 iterations:

descendants ps spawns in a 1500 ms budget cost at 10.9 ms/spawn
1 60 ~0.65 s
4 240 ~2.6 s
8 480 ~5.3 s

The teardown could not meet its own deadline for a tree of four, on an idle
machine.
Reading ps more carefully does not touch that: the cost is the
spawn.

What Node can do, and what it cannot

The Rust tool can replace ps with syscalls. Node cannot, without a native
module.
I considered that and rejected it: a globally installed CLI that needs
a compiler at install time, or prebuilt binaries per platform × architecture ×
Node ABI, buys a smaller bug than it costs. Existing npm packages mostly spawn
ps themselves, which moves the defect behind a dependency.

So this does the two things Node can do:

On Linux there is now no subprocess at all. /proc/<pid>/stat carries ppid,
pgid, state and starttime — every fact the callers ask for.

On macOS ps stays, read once per operation rather than once per process
per poll. In the teardown loop that is 240 spawns down to 60.

Production ps call sites: 6 → 3, and none of the three is inside a
per-process poll loop.

where what it is
proc-table.ts the one table read, off Linux only
server.ts rss/pcpu for one session's stats, off Linux only
recovery.ts the on-disk token contract — see below

The defect that only a cross-platform run could find

An unreaped descendant kept its /proc row and its identity on Linux, so the
teardown counted a corpse as a survivor.
It waited out the whole TERM budget
for a process that could not respond, then reported it as having survived a
SIGKILL. That is a dishonest kill in the other direction — the thing the
kill PR exists to remove, pointing the opposite way.

It was pre-existing, not introduced by this work. The old code matched on a
start token read from /proc, which a zombie still has.

macOS never had it, because libproc drops the corpse the moment it exits —
the same libproc behaviour that is the bug being fixed above. So:

  • neither tool alone would have shown it, because both had it;
  • neither platform alone would have shown it, because each looks correct on its own;
  • the two disagreeing is what exposed it.

Silber.pty ran both branches against the same zombie on the same Mac at the
same moment, got a Z row from one and NotPresent from the other, and
reported the disagreement rather than either result. Chasing why is what found
this.

A control confirms it is a finding rather than a claim: revert the fix and the
test fails with an exited but unreaped child still reads as a live descendant.

What a real Mac has already proved

Silber.pty ran the gate on the only Mac holding both branches:

  • the kill fix works there: two TERM-to-KILL cycles in 3.56 seconds with
    zero ps calls;
  • the registry contract holds: both tools produced
    darwin:Thu Sep 3 13:35:16 2026, double space intact;
  • live ppid, pgid, status and start time from libproc matched /bin/ps
    exactly.

Outstanding on that gate: libproc refuses an unreaped child
(PROC_PIDTBSDINFO returns 0 of 136 bytes with ESRCH) while proc_listpids
still lists it. KERN_PROC_PID answers. That fix is in progress and the gate
will be rerun against it.

What this costs on macOS, stated so nobody finds it later

Silber.pty measured the socket-owner kill proof on a Mac: two kills plus both
starts in 4.16 seconds, making 120 ps calls — 115 whole-table reads and 5
lstart token reads
.

That is option d working as designed, and a large improvement on one spawn per
process per poll. It is still 115 against the Rust tool's zero. Closing that
gap needs a native module, which for a globally installed CLI costs more than
the bug. The number is here so it is a known trade rather than a discovery.

Silence is a third answer

Every query returns an Answer, separating the fact from not-present — the
table was read and this process was not in it — from unknown. No default,
no direct unwrap, no conversion that loses the distinction.
A caller that
wants silence to mean death calls orAbsentWhenUnknown, which is long on
purpose and greps in one command.

This does not make the mistake impossible, and I would rather say so than
imply otherwise.
It makes it visible in review.

A listing that does not contain the process that read it was truncated, not
empty.
ps always lists at least itself. That one comparison turns a silent
or half-written listing into unknown instead of "nothing exists". A control
run confirms it bites: remove the check and two tests fail.

The one ps that must stay, and why

recovery.processStartToken is written into session metadata and read back by
the Rust tool from the same registry
. Its exact text — including the two
spaces ps -o lstart= puts before a single-digit day — is a contract between
two programs, not an implementation detail.

My first parser re-joined whitespace-split fields and would have quietly
rewritten Wed Sep 3 as Wed Sep 3. Nothing would have failed until a Mac
upgraded and stopped recognising its own sessions.
The parser now takes the
tail verbatim, and the in-memory identity is a separate branded type so the
two cannot be compared by accident.

That call is one per session lookup, never in a loop, and its failure already
means "cannot confirm" rather than "gone". It is documented in place.

What is tested, and what is not

1655 passed, 2 failed. Both failures are in tests/shells.test.ts, one for
fish and one for zsh; neither shell is installed on the machine I ran on,
and running that file on unmodified main gives the same two failures and no
others.

12 new tests pin the three answers, the truncation guard, the verbatim lstart
text, the /proc parser against a comm containing spaces and brackets, and —
against the real machine — this process, a pid that cannot exist, and a real
zombie
, which must read as present but not running.

Not tested here: the macOS path. Every test above ran on Linux, where the
ps reader is not the one in use. The ps parser itself is covered by the
listing tests, which are platform-independent, but no macOS machine has run
this code
and it should be exercised on one before merging.

Every caller that needed a fact about a process ran its own `ps` and treated
the output as fact. A subprocess can be slow, truncated or silent, and all
three look exactly like "the process is gone".

Node cannot make the syscalls the Rust tool uses, so this does what Node can.
On Linux there is now no subprocess at all: `/proc` carries ppid, pgid, state
and starttime, which is every fact the callers ask for. On macOS `ps` stays,
but the table is read once per operation rather than once per process per
poll. In the teardown loop that is the difference between 240 spawns inside a
1500 ms deadline and 60.

Silence is a third answer everywhere. Every query separates the fact from "the
table was read and this process is not in it" from "I could not find out",
with no default and no conversion that turns the last into the middle by
accident. Treating silence as death requires calling
`orAbsentWhenUnknown`, which greps in one command. That makes the mistake
visible rather than impossible.

A listing that does not contain the process that read it was truncated, not
empty. `ps` always lists itself.

`recovery.processStartToken` is untouched and still comes from `ps -o lstart=`.
Its exact text is a contract with the Rust tool through a shared registry, so
the parser takes the tail verbatim rather than re-joining split fields, which
would have rewritten `Wed Sep  3` as `Wed Sep 3`. The in-memory identity is a
separate branded type so the two can never be compared.

Production `ps` call sites: six to three, none in a per-process poll loop.
An unreaped descendant keeps its `/proc` row and its identity on Linux, so
matching on identity alone counted it as alive. The teardown would wait out its
whole TERM budget for a process that had already died, and then report it as
having survived a SIGKILL. That is the kill over-claiming again, in the other
direction.

macOS never had this, because `ps` stops listing a process the moment it exits.
The two platforms disagreeing is what exposed it.

Two test fixes from a real Mac run, reported by Silber.pty:

macOS has the `setsid` system call but no `setsid` executable. The real
process-group test spawned the binary, so on the one platform where process
groups are the whole escalation story, the test could not run at all. It now
uses `detached: true`, which is the same thing without the command.

The zombie test asserted the Linux mechanism rather than the conclusion. On
Linux the corpse keeps a row with state Z; on macOS it is dropped from the
listing at once. The test now asserts what every caller depends on, which is
the same on both.

A single-pid query no longer reads the whole table. `hasProcessExitedForReap`
sits inside poll loops that run every 25 ms, and asking about one pid should
not pay for every process on the machine.
The single-process read handed the subprocess's raw stdout to the single-line
parser. `ps` ends its output with a newline and that parser's `$` does not match
before one, so on macOS a live process read as "field-empty" and a zombie read
as NOT EXITED. The teardown would then have waited out its whole budget for a
corpse.

That is the corpse defect a third time, reintroduced by a second parsing path
that Linux never exercised, because on Linux this read goes to `/proc`.

There is one parser now. The single-process read goes through
`parsePsListing`, which splits lines first and checks that the row for the pid
it asked about is actually present, so both jobs are done by the code that was
already tested.

The other two `ps` call sites were checked for the same seam. `server.ts` and
`recovery.ts` both trim before parsing; only the path added by this branch did
not.

Nothing caught it because every test built its input the way the parser
expected. They all agreed with each other and none of them agreed with `ps`. So
there is now a test that runs the real command with the real arguments and feeds
the parser exactly what the subprocess wrote, newline and all. Reverting to the
old behaviour fails it.

Found on a real Mac by Silber.pty.
@myobie
myobie merged commit 589b13d into kill-escalates Sep 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant